What

Adds a GPU test suite that runs in the merge queue and blocks the merge if it fails. It rents an RTX 5090 on Vast.ai, runs the CUDA tests there, and always destroys the box — reusing the rent → provision → run → always-destroy orchestration from benchmark-gpu.yml.

Why

CPU CI (pr_main.yaml) runs on GitHub ubuntu-latest runners, which have no GPU, so the CUDA backend is never exercised before merge. This suite runs the GPU-relevant tests on real hardware.

Test groups (run by scripts/gpu_test.sh via Makefile targets)

Groups 1–3 are GPU-only (no CPU equivalent); 4–5 mirror CPU CI's prover jobs but with the CUDA path enabled. All groups run even if one fails; the job fails if any group does.

How it works

• Trigger: merge_group (one GPU rental per merge, not per push) + workflow_dispatch. A failure ejects the PR from the queue.
• The box checks out the merge ref and runs scripts/gpu_test.sh, which:
  • pins cudarc (CUDARC_PIN=cuda-12080) and verifies the GPU toolchain (prints full nvidia-smi),
  • builds the guest ELFs (compile-programs-asm + compile-programs-rust),
  • runs the 5 groups, aggregating failures.
• Machine info is also printed in its own dedicated step (nvidia-smi, nvcc --version, CPU/RAM) before the test run, so the hardware is easy to find without scrolling the large test log.
• Teardown always destroys the instance (--yes, 3× retry, plus a label-based sweep so a cancelled-mid-rent box can't leak).
• Reporting: full stdout+stderr in the step log; the job summary lists the box specs, the failed tests grouped by suite, and the panic/assertion message for each.

Provisioning: price bands + retry across offers

The provisioning is a single Provision instance (retry across offers) step (ideas borrowed from IDP's create-vast-server.yml), designed so a slow or flaky host can't stall the merge gate:

• Escalating price bands — PRICE_THRESHOLDS: "0.6 0.8 1.0" ($/hr, ascending). It picks the priciest (most reliable) offer in the cheapest band that has capacity, only climbing to a pricier band when a lower one is empty. The last value ($1) is the hard cap.
• Retry across distinct offers — up to MAX_TRIES: 5 different physical hosts. Each host gets a PROVISION_TIMEOUT: 600s readiness budget (running + sshd accepting our key + onstart bootstrap finished); if it isn't ready in time, the box is destroyed and a different offer is tried.
• Tried-host exclusion — every attempted host's machine_id is tracked and excluded from later scans, so a flaky host is never re-picked.
• Fast-fail on unrecoverable states — an offer whose status_msg reports an image that can't be pulled / a host that can't meet the ask ("not started loading", "cannot be met", daemon errors) is abandoned immediately instead of burning the full 600s budget.
• Transient scarcity — within each attempt, all bands are re-scanned up to OFFER_ATTEMPTS: 10 times (30s apart) before giving up, to ride out the small, fast-churning RTX 5090 pool.
• Offer filter: RTX 5090, ≥16 cores, ≥48 GB RAM, ≥64 GB disk, cuda_max_good>=13.1, driver major ≥ 580.
• Each swapped-out box is destroyed in-step (its recorded id cleared), and the final ready box is recorded for the always-run teardown — so no box leaks across the retry loop.

CUDA version requirement

An NVIDIA driver supporting CUDA ≥ 13.1 is required: the kernels are compiled with the toolkit's nvcc into PTX that the driver JIT-compiles at load — an older driver rejects it with CUDA_ERROR_UNSUPPORTED_PTX_VERSION. Enforced by the cuda_max_good>=13.1 offer filter and documented in the README "GPU Tests" section + crypto/math-cuda/build.rs.

Files

• .github/workflows/gpu-tests.yml — new workflow (job gpu-tests).
• scripts/gpu_test.sh — new on-box runner (cudarc pin + compile + 5 groups, aggregate exit).
• Makefile — adds test-cuda-fallback, test-prover-cuda, test-prover-comprehensive-cuda (groups 1 & 2 targets already existed).
• README.md — GPU Tests section + targets table.
• crypto/math-cuda/build.rs — comment clarifying the PTX/driver version coupling.

Before this gates merges (manual GitHub settings — admin, not code)

on: merge_group: only makes the workflow eligible to run in the queue.

1. Add gpu-tests to the required status checks for main (Settings → Branches/Rulesets). Otherwise the job runs in the queue but a failure won't block the merge. Current required checks are only Lint and Test.

Sequencing: the gpu-tests check usually needs to have run once (via workflow_dispatch, or after this merges to main) before it appears in the required-checks picker. merge_group runs the workflow from the merge-commit ref (base + PR), same pattern as the existing merge-queue-only test-prover-comprehensive job.

Repo secrets VAST_API_KEY and VAST_TEMPLATE_HASH are already set.

Status

All 5 groups pass end-to-end on a rented RTX 5090. One real test-harness bug was fixed along the way: test-cuda-fallback ran its two tests in parallel, racing on the process-global GPU dispatch counters (assert_eq! count mismatches) — fixed by adding --test-threads=1 to that Makefile target.

Notes

• Tradeoff: as a merge gate, a persistent Vast outage (no ready box after MAX_TRIES offers across all price bands) would block merges until it clears; the price-band climb, per-host readiness budget + retry-across-offers, and re-scan-for-scarcity loops mitigate it.